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DEFINITIONS 

IDA  publishes  the  following  documents  to  report  the  results  of  its  work. 


Reports 

Reports  are  the  most  authoritative  and  most  carefully  considered  products  IDA  publishes. 
They  normally  embody  results  of  major  projects  which  (a)  have  a  direct  bearing  on 
decisions  affecting  major  programs,  (b)  address  issues  ol  significant  concern  to  the 
Executive  Branch,  the  Congress  and/or  the  public,  or  (c)  address  issues  that  have 
significant  economic  implications.  IDA  Reports  are  reviewed  by  outside  panels  of  experts 
to  ensure  their  high  quality  and  relevance  to  the  problems  studied,  and  they  are  released 
by  the  President  ot  IDA. 

Group  Reports 

Group  Reports  record  the  findings  and  results  of  IDA  established  working  groups  and 
panels  composed  of  senior  individuals  addressing  major  issues  which  otherwise  would  be 
the  subject  of  an  IDA  Report.  IDA  Group  Reports  are  reviewed  by  the  senior  individuals 
responsible  for  the  project  and  others  as  selected  by  IDA  to  ensure  their  high  quality  and 
relevance  to  the  problems  studied,  and  are  released  by  the  President  of  IDA. 

Papers 

Papers,  also  authoritative  and  carefully  considered  products  ot  IDA.  address  studies  that 
are  narrower  in  scope  than  those  covered  in  Reports.  IDA  Papers  are  reviewed  to  ensure 
that  they  meet  the  high  standards  expected  of  refereed  papers  in  professional  journals  or 
formal  Agency  reports. 

Documents 

IDA  Documents  are  used  for  the  convenience  of  the  sponsors  or  the  analysts  (a)  to  record 
substantive  work  done  in  quick  reaction  studies,  (b)  to  record  the  proceedings  ol 
conferences  and  meetings,  (c)  to  make  available  preliminary  and  tentative  results  of 
analyses,  (d)  to  record  data  developed  in  the  course  of  an  investigation,  or  (e)  to  forward 
information  that  is  essentially  unanalyzed  and  unevaluated.  The  review  ol  IDA  Documents 
is  suited  to  their  content  and  intended  use. 
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PREFACE 


The  purpose  of  IDA  Paper  P-2266,  An  Issue  Paper  for  a  Strategic  Defense  Initiative 
Organization  Software  Test  and  Evaluation  Policy,  is  to  describe  the  goals  and  proposed 
contents  of  a  software  test  and  evaluation  (T&E)  policy  for  the  Strategic  Defense 
Initiative  Organization  (SDIO).  Such  a  policy  would  enable  the  SDIO  to  obtain  consistent 
results  from  software  testing  and  evaluation  (T&E),  obtain  the  maximum  benefit  from 
available  T&E  technology,  and  provide  support  for  a  Full-Scale  Development  decision. 

This  document  partially  fulfills  the  objective  of  Task  Order  T-R5-597.21,  SDS  Test 
and  Evaluation,  by  providing  draft  guidelines  for  the  integration  of  test  and  evaluation 
conchas  throughout  the  software  life  cycle.  P-2266  will  be  used  to  identify  the  goals  and 
contents  of  a  software  T&E  policy  for  the  SDIO  and  is  directed  towards  SDIO  and 
element  program  managers. 

The  document  was  reviewed  on  August  11  and  18,  1989,  by  the  members  of  the 
following  CSED  Peer  Review:  Cy  D.  Ardoin,  James  Baldo,  Herbert  R.  Brown,  Cathy  Jo 
Linn,  Katydean  Price,  and  Richard  Wexelblat. 


1.  INTRODUCTION 


This  paper  describes  the  requirements  for  a  Strategic  Defense  System  (SDS) 
Software  Test  and  Evaluation  (T&E)  Policy.  The  T&E1  of  SDS  software  is  one  of  the  key 
factors  affecting  the  creation  of  a  reliable  SDS  [1].  Currently,  little  DoD  guidance  is 
available  to  SDS  contractors  in  this  area.  Although  the  software  annex  of  the  SDS 
Capstone  Test  and  Evaluation  Master  Plan  (TEMP)  [2]  and  the  element  TEMPs  provide 
plans  relating  test  activities  to  required  system  characteristics,  these  planning  documents 
are  not  appropriate  places  for  providing  the  necessary  T&E  technology  guidance. 
Instead,  a  single  authoritative  source  of  T&E  technology  guidance  which  augments  the 
various  TEMPs  is  required.  The  SDS  Software  T&E  Policy  will  fill  this  need. 

In  accordance  with  the  SDS  Software  Policy  [3],  the  Software  T&E  policy  will 
provide  a  strategy  to  promote  a  change  in  attitudes,  policies,  and  practices  concerning 
software  testing.  It  will  create  and  foster  an  aquisition  and  management  environment 
which  encourages,  promotes,  and  rewards  the  use  of  modern  software  testing  practices  in 
the  development  of  mission-critical  development  SDS  software.2 

The  policy  itself  will  be  restricted  to  identifying  requirements  for  software  testing 
practices.  The  services  and  other  implementing  agents  shall  develop  implementation 
documents  that  follow  SDIO  guidelines  set  forth  therein.  These  documents  shall  be 
consistent  with  existing  or  planned  software  engineering  practices.  The  SDIO  will 

1.  Within  this  paper,  the  word  “software”  is  an  implied  prefix  to  the  abbreviation  “T&E”,  unless  otherwise 
qualified. 

2.  It  must  be  remembered  that  mission-critical  SDS  software  includes  simulations,  research,  and  support 
software  in  addition  to  operational  software. 


implement  the  policy  for  its  own  software  development  efforts,  thus  providing  guidance 
for  service  implementation.  The  policy  will  be  reviewed  annually  or  as  needed  against 
evolving  technology. 

2.  GOALS  OF  THE  SOFTWARE  T&E  POLICY 

The  goals  of  the  policy  can  be  grouped  into  three  general  areas. 

2, 1  Provide  consistency  among  diverse  T&E  efforts 

Several  different  groups  will  be  involved  in  SDS  software  T&E.  For  example,  the 
SDS  System  Engineer  will  specify  system  requirements  and  conduct  system  integration 
and  testing  whereas  the  SDS  element  program  offices  are  responsible  for  element-level 
testing.  In  addition  to  those  who  actually  perform  T&E,  there  are  various  organization 
which  support  these  activities.  Examples  in  this  case  include  the  National  Test  Bed  and 
the  Software  Center.  It  is  essential  that  each  group  understand  the  information  and  data 
provided  by  other  groups.  Moreover,  consistency  is  a  prerequisite  for  allowing  related 
activities  to  cooperatively  provide  confidence  in  the  reliability  and  suitability  of  the  SDS 
without  unnecessary  duplication  of  effort. 

The  policy  must  fulfill  two  major  objectives.  First,  it  must  provide  for  uniform 
expression  of  explicit,  traceable  T&E  requirements.  These  requirements  must  support 
common  interpretation  by  different  groups  and  enforce  at  least  a  necessary  minimum 
level  of  testing.  Second,  the  policy  must  ensure  consistent  reporting  of  T&E  results.  Not 
only  must  they  clearly  indicate  the  success  or  failure  of  particular  tests,  but  T&E  results 


must  also  provide  unambiguous  information  about  the  status  of  the  software. 
Furthermore,  this  information  must  provide  feedback  to  both  the  development  and  T&E 
activities.  Failure  to  achieve  these  objectives  will  severely  limit  the  ability  to  demonstrate 
confidence  in  the  reliability  of  the  SDS. 

2.2  Obtain  the  maximum  benefit  from  available  T&E  technology 

Traditionally,  T&E  is  used  simply  as  an  acceptance  challenge  at  the  end  of  each 
development  phase.  Such  an  approach  fails  to  exploit  the  potential  of  available  T&E 
technology.  Not  only  does  late  identification  of  errors  adversely  affect  development  costs 
and  schedules,  but  reliability  cannot  be  tested  into  software  Consequently,  the  policy 
must  ensure  that  T&E  is  an  integral  part  of  development  activities,  not  a  final  step  to  “get 
the  bugs  out.” 


Another  key  point  to  be  addressed  by  the  policy  is  automated  support  of  T&E.  Tools 
to  support  T&E  planning,  performance,  analysis,  and  reporting  are  not  merely  desirable 
but  increasingly  indispensable.  Indeed,  many  advanced  testing  techniques  cannot  be 
applied  in  the  absence  of  suitable  automated  support.  The  current  lack  of  production 
quality  tools  is  one  of  the  contributing  factors  to  the  extreme  lag  of  current  testing 
practice  behind  the  state  of  the  art.  The  policy  must  address  the  responsibilities  and 
resources  needed  to  ensure  the  availability  of  appropriate  automated  support  to  T&E 
performers. 


2.3  Support  a  Full-Scale  Development  (FSD)  decision 


A  crucial  question  in  making  the  SDS  FSD  decision  will  be  whether  the  necessary 
confidence  in  the  correct  and  reliable  operation  of  the  software  can  be  achieved,  and  at 
what  cost  this  confidence  can  be  obtained.  If  the  decision  to  proceed  is  made,  then  the 
experience  gained  during  previous  T&E  activities  should  be  exploited  in  planning  tot 
FSD  T&E.  The  key  issue  is  the  need  for  better  understanding  of  the  capabilities  ot 
available  technology  and  how  this  understanding  may  be  used  to  reason  about  the 
software.  This  understanding  should  be  used  to  support  both  T&E  and  development 
activities. 

The  policy  should  specify  the  actions  and  responsibilities  necessary  for  collecting  and 
analyzing  data  on  the  costs  and  capabilities  of  specific  T&E  and  development  practices. 
It  should  also  specify  how  the  accumulated  information  will  be  fedback  to  improve  T&E 
and  development  practices.  With  respect  to  the  FSD  decision,  information  on  the 
capabilities  of  T&E  technology  should  be  interpreted  in  the  light  of  trends  in  the  practical 
experience  with  dcmonstration/validation  software  and  the  history  of  technology 
development. 

3.  REQUIREMENTS  OF  THE  SOFTWARE  T&E  POLICY 

In  general,  the  policy  will  be  the  mechanism  for  defining  the  activities  and 
responsibilities  of  the  various  T&E  participants  in  achieving  the  necessary  change  in 
practices  and  attitudes  to  ensure  effective  T&E.  The  following  requirements  will  provide 
an  initial  step  toward  achieving  the  above  goals. 
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3.1  Define  common  terminology 


The  current  lack  of  a  common  SDIO  T&E  terminology  is  a  severe  handicap  in 
acquiring  a  consistent  view  of  the  status  of  the  software  and  advancing  the  state  of  the 
practice.  For  example,  the  Army,  Air  Force,  and  Navy  all  have  different  definitions  for 
software  reliability.  The  policy  must  provide  a  single,  authoritative  source  for  TAF 
terminology  which  applies  to  all  SDS  software  development  efforts. 

3.2  Integrate  testing  activities  into  development 

To  ensure  that  the  necessary  confidence  in  the  software  can  be  acquired  (and  at 
acceptable  cost),  software  testing  issues  must  be  considered  during  the  earliest 
development  stages.  As  system  requirements  are  allocated  to  software,  explicit  testing 
requirements  must  be  identified  in  conjunction  with  functional  and  performance 
requirements.  Not  only  must  these  testing  requirements,  along  with  the  functional  and 
performance  requirements,  be  traceable  through  all  development  stages,  but  they  must 
specify  the  types  and  extent  of  necessary  testing  for  specific  development  products  at 
each  development  stage.  Additionally,  as  previously  noted,  the  actual  testing  process 
must  not  be  postponed  until  the  final  development  stages.  Indeed,  in  many  cases  (e.g.. 
formal  verification)  the  technology  can  only  be  effectively  applied  as  an  integral  part  of 
development  activities.  Consequently,  SDS  T&E  must  provide  software  developers  with 
timely  information  that  they  can  act  on  to  ensure  the  development  of  reliable  and  quality 
products. 

One  approach  for  ensuring  proper  integration  of  testing  into  development  activities 
will  be  to  specify  test  articles  throughout  the  life  cycle,  along  with  the  test  requirement:. 


s 


which  determine  the  testing  to  be  performed  on  those  articles.  The  policy  will  provide 
suitable  me  'hanisms  for  requiring  the  timely  specification  of  such  requirements  and  test 
articles  and  also  for  ensuring  that  the  necessary  test  events  are  planned  for  and 
conducted.  For  example,  the  policy  may  make  testing  requirements  a  deliverable 
product.  It  is  anticipated  that  the  SDlO-implementing  directive  will  provide  an  initial 
syntax  and  semantics  for  the  testing  requirements  and  specify  the  conduct  of  formal 
reviews  to  assure  their  appropriate  use. 

In  addition  to  providing  a  eommon  mechanism  for  determining  the  minimum  levels 
of  required  testing  for  all  development  products,  testing  requirements  will  Drovide  a 
foundation  for  T&E  resource  estimation  and  scheduling  during  both  development  and 
support  phases.  The  policy  will  provide  mechanisms  for  requiring  timely  resource 
estimation  and  scheduling  while  the  SDIO  implementing  directive  will  likely  specify 
suitable  technology  for  performing  these  activities. 

It  is  recommended  that  validation  suites  (similar  in  nature  to  those  used  in  the  Ada 
Compiler  Validation  Capability  (ACVC)  [4])  pertaining  to  particular  testing  requirement 
and  development  product  combinations  be  evolved  in  the  course  of  T&E  activities.  As 
demonstrated  by  the  ACVC,  the  advantages  offered  by  such  validation  suites  are 
considerable.  One  is  the  ability  to  react  to  the  discovery  of  an  error,  and  tracing  it  back 
through  a  series  of  earlier  development  products  if  necessary.  This  ability  will  be 
invaluable  for  a  system  with  the  size  and  requirements  evolvability  of  SDS.  The  policy 
will  establish  the  appropriate  mechanisms  for  requiring  the  development  and  use  of 
validation  suites,  together  with  assigning  responsibilities  for  the  establishment  and 
maintenance  of  necessary  facilities. 
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3.3  Define  information  and  data  requirements  for  T&E  activities 


The  prime  mechanism  for  ensuring  consistency  of  T&E  information  will  be  the 
specification  of  explicit  data  requirements.  These  data  requirements  must  be  designed  to 
ensure  the  following: 

•  All  pertinent  issues  will  be  considered  during  T&E  planning  and  that  planning 
documents  will  include  all  the  information  needed  to  provide  sufficient  guidance  for 
actually  conducting  T&E  activities.  For  example,  test  plans  for  particular  test  events 
must  explicitly  specify  test  completeness  criteria. 

•  1'esting  activities  will  provide  the  data  required  for  analyzing  the  T&E  process  and 
determining  its  effectiveness  and  adequacy. 

•  Test  results  will  facilitate  accurate  understanding  and  communication  of  the  software 
status. 

•  Test  histories  will  facilitate  efficient  regression  testing. 

The  policy  will  impose  a  mechanism  for  attaining  the  necessary  data  from  T&E 
participants.  It  will  also  assign  responsibilities  for  establishing  and  maintaining  a  T&E 
database. 

3.4  Support  a  technology  information  base 

Information  which  can  guide  the  selection  and  application  of  available  techniques 
and  tools  in  all  aspects  of  T&E  planning,  performance,  analysis,  and  reporting  is  needed. 
Establishing  a  technology  database  is  not  within  the  scope  of  the  policy — this  is  within  the 
purview  of  the  Software  Center.  However  the  policy  should  establish  mechanisms  which 


require  T&E  participants  to  provide  the  necessary  data  and  which  encourage  appropriate 
use  of  the  information  base. 

3.5  Support  for  a  tool  repository 

The  encouragement  of  advanced  testing  practices  must  be  supported  by  the  provision 
of  necessary  automated  tools.  As  in  the  case  of  the  technology  information  base,  the 
development  of  a  tool  repository  does  not  fall  under  the  purview  of  an  T&E  policy. 
However,  the  policy  will  identify  who  is  responsible  for  establishing  and  maintaining  the 
repository  and  provide  mechanisms  which  assure  the  appropriate  use  of  the  available 
tools  by  T&E  participants. 

4.  SUPPORTING  DEVELOPMENT  OF  THE  SOFTWARE  T&E  POLICY 

Five  actions  must  be  performed  either  prior  or  in  conjunction  with  development  of 
the  policy: 

4.1  Establish  contacts  with  T&E  participants  and  assess  current  practices 

It  is  necessary  to  determine  (1)  what  T&E  practices  are  currently  in  use,  or  are 
planned  for  use,  and  (2)  the  quality  of  these  practices.  This  analysis  will  constitute  a 
baseline  of  the  relevant  state  of  the  practice  which  will  assist  in  determining  the  necessary 
extent  and  rate  of  technology  transition. 

4.2  Discuss  intent  of  policy  with  SDIO  T&E  organizations  and  T&E  participants 

The  SDIO  Test  and  Evaluation  Working  Group  (TEWG)  has  recently  established  a 
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Software  Subcommittee.  This  subcommittee  will  be  the  primary  interface  between 
prospective  T&E  participants  and  the  developers  of  the  policy.  One  of  the  objectives  of 
initial  meetings  of  the  subcommittee  will  be  to  agree  on  the  needs  and  objectives  of  the 
policy  and  solicit  initial  support  for  activities  and  responsibilities.  The  relationships  and 
roles  of  the  various  groups  in  T&E  planning,  conduct,  reporting,  and  the  various  support 
activities  previously  outlined  must  be  clarified  with  the  relationships  and  roles  of  testing 
agents  and  developers. 

4.3  Coordinate  policy  with  other  SDIO  policies  and  directives,  and  with  applicable  DoD 
standards 

It  may  be  necessary  to  provide  tailoring  of  established  standards. 

4.4  Increase  awareness  of  available  technology 

There  are  several  advanced  T&E  technologies  that  are  ready  for  transition  into 
practice  [5].  Nevertheless,  before  these  techniques  can  be  mandated  for  use,  hard  data  is 
needed  on  their  benefits  and  cost  along  with  practical  experience  in  their  use.  A  series  of 
demonstrations  of  suitable  technologies  should  be  undertaken. 

4.5  Develop  a  technology  research  plan  to  address  critical  gaps  in  T&E  technology 

Many  of  the  T&E  challenges  being  faced  by  SDS  also  impact  other  large  scale 
software  development  efforts.  These  challenges  are  recognized.  Promising  research  is 
being  conducted  by  academia,  for  example,  the  goal-directed  measurement  work  at  the 
University  of  Maryland  [6],  and  organizations  such  as  National  Aeronautics  Space 


Administration  (NASA)  [7].  The  SDIO  should  establish  and  maintain  contact  with  these 
research  communities  to  ensure  that  best  use  is  made  of  available  research  dollars  in 
attacking  critical  technology  deficiencies. 

In  particular,  the  SDIO  should  support  the  development  of  a  comprehensive  testing 
environment.  While  a  set  of  individual  tools  to  support  advanced  testing  techniques  can 
be  made  available  within  the  near  future,  the  SDIO  should  work  towards  providing  an 
integrated  testing  environment.  This  environment  should  provide  a  wide  range  of  testing 
and  analysis  techniques,  together  with  supporting  tools  such  as  test  drivers,  test  data 
generators,  and  reporting  mechanisms.  It  must  also  support  the  data  collection  and 
analysis  activities  called  for  in  the  policy  requirements.  Coordination  with  the  software 
engineering  environment  efforts  by  the  SDIO  Software  Center  will  enable  development 
activities  to  be  integrated  with  testing  activities.  In  addition,  there  are  a  number  of 
advanced  design  principles  which  are  prerequisites  for  a  suitable  environment: 

•  An  evolutionary  architecture  which  facilitates  growth  as  new  testing,  analysis  or 
support  techniques  are  developed  (e.g.,  a  tool  fragment  approach  [8]). 

•  Persistent  object  management  [9,10]  which  allows  an  evolving  collection  of  tools  to 
be  applied  to  test  objects. 

•  Incremental  analysis  capabilities  so  that  testing  and  analysis  can  be  performed  on 
incomplete  software  products. 

•  Integrated  application  of  testing  and  analysis  techniques  [11], 

Additionally,  the  use  of  process  proprams  which  provide  precise  and  rigorous 
description  of  software  processes  [12,13]  should  be  investigated  as  a  possible  mechanism 
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for  supporting  these  capabilities  and  providing  developers  with  pro-active  guidance  in 
testing  activities. 
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